1. Technical Field
The invention is related to construction of panoramic images, and in particular, to an image capture device, such as a digital still or video camera, which provides real-time interactive assistance to a user in capturing images used in constructing the panoramic images.
2. Related Art
In general, a mosaic panoramic image is an image that has been expanded in one or more dimensions by combining multiple overlapping image frames to generate the panorama. As a result, the panoramic image is typically larger than the size that a conventional image capture device (such as a digital still or video camera) is capable of capturing in a single image frame.
A number of conventional techniques have been developed to generate such panoramic images. For example, the basic approach is to simply take a plurality of regular photographic or video images in order to cover the entirety of the desired viewing space. These images are then aligned and composited into complete panoramic images using any of a number of conventional image “mosaicing” or “stitching” algorithms.
Early stitching schemes generally required a carefully controlled camera motion resulting in a known overlap between images. However, more recent stitching schemes have allowed users to take photos covering a random 2D field with pictures taken in an arbitrary order. In either case, as is well known to those skilled in the art, such systems generally take a set of images captured by the digital still or video camera and perform post-processing operations on those images, including, for example, aligning, compositing, deghosting, blending of overlapped areas, automatic adjustments of color, brightness, contrast, etc., to generate a mosaic image which is then typically cropped to a rectangular region or frame to create the final panoramic image.
However, one problem with such post processing of images is that the user never really knows for sure that sufficient coverage of the scene has been achieved to ensure that the desired panorama can be constructed from the set of captured image frames until the panorama has actually been generated at some later time. As a consequence it can be hard for users to see “the big picture.”
In particular, using conventional image stitching schemes, it is not until the images are uploaded from the camera to the computing device (such as a PC-type computer, or the like) that users find that the resulting panorama is flawed. For example, gaps will occur in the panorama if users miss one or more spots. Gaps will also occur if the stitching program is unable to insert one or more of the captured images into the panorama, e.g., due to too little overlap between pictures, due to a lack of texture, due to problems with image focus (image too blurry), etc. Further, while gaps at the edge of a panorama can be removed by cropping the panorama, this may cause other desired image elements to be lost outside the cropping boundaries. In the worst case, what was intended to be a single panorama can end up as multiple unconnected (and odd-shaped) pieces of the overall image.
Another problem with conventional post-processing approaches involves the issue of “ghosting” where objects have moved from one frame to the next while the images for the panorama were being taken. While users may be able to identify flaws within a picture (e.g., out of focus) using the view finder, and retake it, flaws between photos, such as ghosting or differences in lighting etc. are less likely to be discovered while shooting. Unfortunately, by the time users notice such flaws, it is typically too late to retake the missing or flawed image frames.
Users can reduce the risk of flaws such as gaps and ghosting by using a larger overlap between images and by taking multiple copies of areas suspected of being subject to ghosting. This redundancy-based approach, however, is costly is terms of time and especially in terms of storage space (as more pictures are taken) and it still cannot guarantee successful generation of the resulting panoramic images.
At least one conventional scheme partially addresses some of these issues. For example, one conventional scheme, referred to as “VideoBrush™ ” provides a near-real-time preview of a panoramic image constructed from images captured by a video camera. However, the VideoBrush™ system supports only translational motion, not full-view panoramas. In particular, the VideoBrush™ system provides techniques for 1 D and 2D video mosaicing using parametric alignment which includes videos captured with an approximately fixed camera location, or an arbitrarily moving camera capturing an approximately planar scene. Further, another problem with systems such as that offered by VideoBrush™ is that they do not provide explicit indications for issues such as whether desired scene elements will survive cropping to a conventional rectangular format, or whether the resulting panorama will be subject to ghosting.
Similarly, some digital consumer cameras offer a “stitch assist mode,” or the like, in which the LCD (or other electronic display) shows the camera “viewfinder” and a part of the last frame taken for visual reference. For example, digital cameras such as the “Canon Powershot S230” provide an approximately 1D panoramic stitch assistance which requires the user to aim the camera, such that the viewfinder lines up with a reference image (typically the last image captured) displayed in a sub-region of the viewfinder. The latest frame then becomes the reference for the next shot, and so on. This approach guarantees successful stitching by ensuring the right amount of overlap. Also, the known spatial relationship between frames can simplify final post-hoc stitching of the images using a separate computing device.
Unfortunately, the approach offered by such cameras is limited in several ways. First, such approaches do not extend well to 2D panoramas. In particular, even if users could choose a connection direction between images as they move along, shooting a large 2D image by brushing back and forth in rows would only guarantee connectivity between predecessor and successor images, but not between the rows themselves. Second, such approaches offer no guarantee that the desired scene elements will survive cropping. Third, such approaches tend to limit the users' freedom, as the user is required to constantly focus on the LCD display of the camera. Further, issues such as ghosting and focus are not adequately addressed by such systems.
A similar system is provided by the “HP Photosmart R707” camera, which offers a stitch assist mode similar to that of the aforementioned “Canon Powershot S230,” so taking panoramic pictures is subject to the same limitations. In addition, however, cameras such as the HP Photosmart R707 allow users to preview the entire panorama on the camera's LCD screen in playback. This allows users to see flaws while still on-site. Unfortunately, fixing such flaws still requires the user to retake the entire panorama, as images can not be inserted into the existing panorama to fix any identified flaws.